Language model customization in speech recognition for speech analytics

ABSTRACT

A method for generating a language model for an organization includes: receiving, by a processor, organization-specific training data; receiving, by the processor, generic training data; computing, by the processor, a plurality of similarities between the generic training data and the organization-specific training data; assigning, by the processor, a plurality of weights to the generic training data in accordance with the computed similarities; combining, by the processor, the generic training data with the organization-specific training data in accordance with the weights to generate customized training data; training, by the processor, a customized language model using the customized training data; and outputting, by the processor, the customized language model, the customized language model being configured to compute the likelihood of phrases in a medium.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. patent application Ser. No.15/247,645, filed in the United States Patent and Trademark Office onAug. 25, 2016, which in turn claims the benefit of U.S. ProvisionalPatent Application No. 62/279,671, filed in on Jan. 16, 2016, the entiredisclosures of which are incorporated herein by reference.

FIELD

Aspects of embodiments of the present invention relate to the field ofsoftware for operating contact centers, in particular, software forperforming speech recognition and analytics on voice interactionsoccurring in a contact center and methods for the automated and assistedconfiguration of such software and services.

BACKGROUND

Generally, a contact center is staffed with agents who serve as aninterface between an organization, such as a company, and outsideentities, such as customers. For example, human sales agents at contactcenters may assist customers in making purchasing decisions and mayreceive purchase orders from those customers. Similarly, human supportagents at contact centers may assist customers in solving problems withproducts or services provided by the organization. Interactions betweencontact center agents and outside entities (customers) may be conductedby speech voice (e.g., telephone calls or voice over IP or VoIP calls),video (e.g., video conferencing), text (e.g., emails and text chat), orthrough other media.

SUMMARY

Aspects of embodiments of the present invention are directed to systemsand methods for automatically converting speech to text and systems andmethods for performing voice analytics in a contact center.

According to one embodiment of the present invention, a method forgenerating a language model for an organization includes: receiving, bya processor, organization-specific training data; receiving, by theprocessor, generic training data; computing, by the processor, aplurality of similarities between the generic training data and theorganization-specific training data; assigning, by the processor, aplurality of weights to the generic training data in accordance with thecomputed similarities; combining, by the processor, the generic trainingdata with the organization-specific training data in accordance with theweights to generate customized training data; training, by theprocessor, a customized language model using the customized trainingdata; and outputting, by the processor, the customized language model,the customized language model being configured to compute the likelihoodof phrases in a medium.

The organization-specific training data may include in-medium data andout-of-medium data.

The in-medium data are speech recognition transcript text and theout-of-medium data may be non-speech text.

The organization-specific training data may include no in-medium data.

The assigning the plurality of weights to the generic training data mayinclude: partitioning the generic training data into a plurality ofpartitions in accordance with the computed similarities; associating apartition similarity with each of the partitions, the partitionsimilarity corresponding to the average similarity of the data in thepartition; and assigning a desired weight to each partition, the desiredweight corresponding to the partition similarity of the partition.

The desired weight of a partition may be exponentially decreasing withdecreasing partition similarity.

The method may further include: receiving organization-specificin-medium data; combining the organization-specific in-medium data withthe generic training data and the organization-specific training data togenerate the customized training data; and retraining the language modelin accordance with the customized training data.

According to one embodiment of the present invention, a system includes:a processor; and memory coupled to the processor and storinginstructions that, when executed by the processor, cause the processorto: receive organization-specific training data; receive generictraining data; compute a plurality of similarities between the generictraining data and the organization-specific training data; assign aplurality of weights to the generic training data in accordance with thecomputed similarities; combine the generic training data with theorganization-specific training data in accordance with the weights togenerate customized training data; train a customized language modelusing the customized training data; and output the customized languagemodel, the customized language model being configured to compute thelikelihood of phrases in a medium.

The memory may further store instructions that, when executed by theprocessor, cause the processor to: receive organization-specificin-medium data; combine the organization-specific in-medium data withthe generic training data and the organization-specific training data togenerate the customized training data; and retrain the language model inaccordance with the customized training data.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, together with the specification, illustrateexemplary embodiments of the present invention, and, together with thedescription, serve to explain the principles of the present invention.

FIG. 1 is a schematic block diagram of a system for supporting a contactcenter in providing contact center services according to one exemplaryembodiment of the invention.

FIG. 2 is a block diagram illustrating a speech recognition engine andtraining modules for a speech recognition engine according to oneembodiment of the present invention.

FIG. 3 is a flowchart of a method for generating a customized languagemodel according to one embodiment of the present invention.

FIG. 4A is a flowchart of a method for extracting organization specifictraining data from a plurality of sources according to one embodiment ofthe present invention.

FIG. 4B is a flowchart of a method for cleaning out-of-medium dataaccording to one embodiment of the present invention.

FIG. 5 is a flowchart of a method according to one embodiment of thepresent invention for filtering out sentences from the out-of-mediumtraining data to remove sentences with spelling errors FIG. 6A is aflowchart of a method for selecting data from the out-of-medium trainingmaterial using an in-medium/out-of-medium cross entropy differencetechnique according to one embodiment of the present invention.

FIG. 6B is a flowchart of a method for computing the weights ofout-of-medium sentences according to one embodiment of the presentinvention.

FIG. 7A is a flowchart of a method for selecting data from out-of-mediumtraining data using cross entropy minimization according to oneembodiment of the present invention.

FIG. 7B is a flowchart of a method for evaluating a sentence fromout-of-medium training data using cross entropy minimization accordingto one embodiment of the present invention.

FIG. 7C is a flowchart illustrating a method for updating thedenominator in accordance with one embodiment of the present invention.

FIG. 7D is a flowchart of a method for evaluating a sentence fromout-of-medium training data represented as bags-of-trigrams according toone embodiment of the present invention.

FIG. 8 is a flowchart of a method according to one embodiment of thepresent invention for combining different training data in accordance tothe relevance of the training data.

FIG. 9A is a block diagram of a computing device according to anembodiment of the present invention.

FIG. 9B is a block diagram of a computing device according to anembodiment of the present invention.

FIG. 9C is a block diagram of a computing device according to anembodiment of the present invention.

FIG. 9D is a block diagram of a computing device according to anembodiment of the present invention.

FIG. 9E is a block diagram of a network environment including severalcomputing devices according to an embodiment of the present invention.

DETAILED DESCRIPTION

In the following detailed description, only certain exemplaryembodiments of the present invention are shown and described, by way ofillustration. As those skilled in the art would recognize, the inventionmay be embodied in many different forms and should not be construed asbeing limited to the embodiments set forth herein. Like referencenumerals designate like elements throughout the specification.

Generally, speech analytics systems have several use cases. In one usecase, speech analytics can be used to classify customer-agentconversations into predefined categories. For example, a customer'sphrases such as “I want to speak to your supervisor” and “I am veryangry” may indicate an event of customer dissatisfaction, and phrasessuch as “thank you for taking care of this” may indicate a successfulresolution. Similarly, an agent's phrases such as “would you like toupgrade to a higher tier plan?” and “may I suggest another model?” mayindicate upselling events and cursing or calling a customer “stupid” oran “idiot” may be grounds for flagging the agent for reprimand orfurther supervisor review.

In another use case, automatically transcribed conversations can besearched for particular terms (e.g., for instances of particular words).In still another use case, new categories or topics can be automaticallydetected from the automatically recognized speech in order to allow anorganization to identify phenomena (potentially new, previouslyuntracked phenomena) that may be occurring during customer-agentinteractions (e.g., in the corpora of recorded interactions).

Due to the difficulties in processing speech from a wide range ofspeakers who may use different pronunciations of words, who may speak atdifferent speeds and at different pitches, and who my speak withdifferent accents, a universal model for accurately recognizing speechcan be very large and may often be shared by many differentorganizations who may have very different businesses (e.g., a pet supplycompany may be very different from a company that sells computersoftware compilers).

However, this universal language model may also be unable to accuratelyrecognize phrases and speech patterns that are particular to certainverticals or industries (e.g., words or phrases relating to petsupplies, such as “kibble,” “aquarium,” and “bedding” versus words orphrases relating to compilers, such as “segfault,” “monad,” and“runtime.”).

In addition, it may be difficult to determine whether a given phraseappears in a transcription of spoken conversations, due to the widevariability in ways that the phrase may be said and due to the poorperformance of the universal speech models on context specific words(e.g., the word “segfault” may be incorrectly recognized as “steaksauce” if the speech recognition system is not tailored for the contextof computer programming).

As such, aspects of embodiments of the present invention are directed tosystems and methods for customizing or tailoring a speech recognitionsystem based on knowledge of the context in which the speech recognitionsystem is applied. For example, knowledge about typical requests of petowners and words and phrases used by pet owners may be provided to aspeech recognition system according to embodiments of the presentinvention in order to customize a speech recognition system expected toencounter interactions involving pet owners (e.g., the contact centerfor a chain of pet stores). This customization allows the speechrecognition system to increase the likelihood of some phrases anddecrease the likelihood of other phrases, and thereby increasing theaccuracy of the output of the speech recognizer. The data for thiscustomization can come from both in-medium data and out-of-medium data.For example, in embodiments of the present invention in which the“medium” is speech (e.g., voice interactions), in-medium training datamay be existing text transcriptions of voice interactions in the contextof the organization (which is generally a small amount of data due tothe expense of generating such transcriptions), and the out-of-mediumdata include textual (or text) data other than text transcriptions ofvoice interactions (e.g., text chats, emails, forum posts, and datascraped from the organization website), and this out-of-medium data maybe referred to as “non-speech text.”

Aspects of embodiments of the present invention are also directed tosystems and methods for supplementing the training data of a speechrecognition system using voice data and transcripts of the voice datafrom sources outside of the organization (also referred to herein asglobal in-medium training materials). For example, recorded voiceinteractions and transcripts of those voice interactions from a varietyof contact centers such as banks, consumer electronics manufacturers,telecommunications providers, retail sales companies, and pharmacies mayall be available for training a speech recognition system. In someembodiments of the present invention, the existing training data may beused to train a speech recognition system for use with a new contactcenter for a particular organization. However, in many instances, thesedata are not tailored to the expected content of the interactions of theorganization. For example, the phrase “bank statement” is unlikely tooccur in interactions with a retail sales company. Therefore, someaspects of embodiments of the present invention are directed to usingexisting training data to customize a speech recognition system in waysthat take into account the expected content of the data that will beencountered by the speech recognition system.

Aspects of embodiments of the present invention may also include: addingcontext specific information to the language model of the speechrecognition system to emphasize the importance of the context specificinformation; searching for context specific topics in the interactions(e.g., large vocabulary continuous speech recognition (LVCSR), chat,email, social media, etc.) of the organization, taking into accountpossible speech recognition errors and acceptable variants of phrases;and incorporating wild cards into the search, so as to enable thediscovery of regular expressions on the text.

Aspects of embodiments of the present invention are also related toadding organization-specific data to the language model in a way thatemphasizes the importance of the organization-specific data. Forexample, the global training material is generally a different size thanthe organization specific training material and/or the context specifictraining material (e.g., the global training material is generally amuch larger data set than the organization specific or context specifictraining material). As such, some aspects of embodiments of the presentinvention relate to systems and methods for assigning weights to thedifferent sources of training material in accordance with their relativesize and in accordance with their relative importance. This techniquemay be referred to herein as “interpolation.” In addition, interpolationmay be used in the customization of the language model described abovefor “time zero” customization of the language model and on-goingcustomization of the language model, in which the organization-specifictraining data includes no in-medium training data (e.g., because theorganization has not been operating a contact center, has not beencollecting in-medium data, or has not collected enough in-mediumtraining data to be worthwhile to include the data).

Contact Center Overview

FIG. 1 is a schematic block diagram of a system for supporting a contactcenter in providing contact center services according to one exemplaryembodiment of the invention. The contact center may also be referred toas a customer contact center. The contact center may be an in-housefacility to a business or enterprise for serving the enterprise inperforming the functions of sales and service relative to the productsand services available through the enterprise. In another aspect, thecontact center may be operated by a third-party service provider.According to some embodiments, the contact center may operate as ahybrid system in which some components of the contact center system arehosted at the contact center premise and other components are hostedremotely (e.g., in a cloud-based environment). The contact center may bedeployed in equipment dedicated to the enterprise or third-party serviceprovider, and/or deployed in a remote computing environment such as, forexample, a private or public cloud environment with infrastructure forsupporting multiple contact centers for multiple enterprises. Thevarious components of the contact center system may also be distributedacross various geographic locations and computing environments and notnecessarily contained in a single location, computing environment, oreven computing device.

For the purposes of the discussion herein, interactions betweencustomers using end user devices 10 and agents at a contact center usingagent devices 38 may be recorded by call recording module 40 and storedin mass storage device 126. The recorded calls may be processed byspeech recognition module 44 to generate recognized text which is storedin the mass storage device 126. In some embodiments of the presentinvention, a voice analytics module 45, may include a text searchmodule, which will be described in more detail below. For the purposesof the discussion herein, interactions between customers using end userdevices 10 and agents at a contact center using agent devices 38 may berecorded by call recording module 40 and stored in the mass storagedevice 126. The recorded calls may be processed by speech recognitionmodule 44 to generate recognized text which is stored in the massstorage device 126. In some embodiments of the present invention, avoice analytics module 45 configured to perform analytics on recognizedspeech data such as by detecting events occurring in the interactionsand categorizing the interactions in accordance with the detectedevents. Aspects of speech analytics systems are described, for example,in U.S. patent application Ser. No. 14/586,730 “System and Method forInteractive Multi-Resolution Topic Detection and Tracking,” filed in theUnited States Patent and Trademark Office on Dec. 30, 2014, the entiredisclosure of which is incorporated herein by reference.

According to one example embodiment, the contact center system managesresources (e.g. agents, support personnel, computers, telecommunicationsequipment, mobile devices, internet-of-things devices, and self-serviceresources) to enable delivery of services via telephone or othercommunication mechanisms. Such services may vary depending on the typeof contact center, and may range from customer service to help desk,emergency response, telemarketing, order taking, and the like.

Customers, potential customers, or other end users (collectivelyreferred to as customers or end users, e.g., end users 106) desiring toreceive services from the contact center may initiate inboundcommunications (e.g., telephony calls) to the contact center via theirend user devices 108 a-108 c (collectively referenced as 108). Each ofthe end user devices 108 may be a communication device conventional inthe art, such as, for example, a telephone, wireless phone, smart phone,personal computer, electronic tablet, and/or the like. Users operatingthe end user devices 108 may initiate, manage, and respond to telephonecalls, emails, chats, text messaging, web-browsing sessions, and othermulti-media transactions.

Inbound and outbound communications from and to the end user devices 108may traverse a telephone, cellular, and/or data communication network110 depending on the type of device that is being used, where thesedevices may include internet-of-things (IoT) devices. For example, thecommunications network 110 may include a private or public switchedtelephone network (PSTN), local area network (LAN), private wide areanetwork (WAN), and/or public wide area network such as, for example, theInternet. The communications network 110 may also include a wirelesscarrier network including a code division multiple access (CDMA)network, global system for mobile communications (GSM) network, or anywireless network/technology conventional in the art, including but tolimited to 3G, 4G, LTE, and the like. Various communications protocolsmay be used on these networks.

According to one example embodiment, the contact center includes aswitch/media gateway 112 coupled to the communications network 110 forreceiving and transmitting telephony calls between end users and thecontact center. The switch/media gateway 112 may include a telephonyswitch or communication switch configured to function as a centralswitch for agent level routing within the center. The switch may be ahardware switching system or a soft switch implemented via software. Forexample, the switch 112 may include an automatic call distributor, aprivate branch exchange (PBX), an IP-based software switch, and/or anyother switch with specialized hardware and software configured toreceive Internet-sourced interactions and/or telephone network-sourcedinteractions from a customer, and route those interactions to, forexample, an agent telephony or communication device. In this example,the switch/media gateway establishes a voice path/connection (not shown)between the calling customer and the agent telephony device, byestablishing, for example, a connection between the customer's telephonydevice and the agent telephony device.

According to one exemplary embodiment of the invention, the switch iscoupled to a call controller 118 which may, for example, serve as anadapter or interface between the switch and the remainder of therouting, monitoring, and other communication-handling components of thecontact center.

The call controller 118 may be configured to process PSTN calls, VoIPcalls, and the like. For example, the communication server 118 may beconfigured with computer-telephony integration (CTI) software forinterfacing with the switch/media gateway and contact center equipment.In one embodiment, the call controller 118 may include a sessioninitiation protocol (SIP) server for processing SIP calls. According tosome exemplary embodiments, the call controller 118 may, for example,extract data about the customer interaction such as the caller'stelephone number, often known as the automatic number identification(ANI) number, or the customer's internet protocol (IP) address, or emailaddress, and communicate with other CC components in processing theinteraction.

According to one exemplary embodiment of the invention, the systemfurther includes an interactive media response (IMR) server 122, whichmay also be referred to as a self-help system, virtual assistant, or thelike. The IMR server 122 may be similar to an interactive voice response(IVR) server, except that the IMR server 122 is not restricted to voice,but may cover a variety of media channels including voice. Taking voiceas an example, however, the IMR server 122 may be configured with an IMRscript for querying customers on their needs. For example, a contactcenter for a bank may tell customers, via the IMR script, to “press 1”if they wish to get an account balance. If this is the case, throughcontinued interaction with the IMR server 122, customers may completeservice without needing to speak with an agent. The IMR server 122 mayalso ask an open ended question such as, for example, “How can I helpyou?” and the customer may speak or otherwise enter a reason forcontacting the contact center. The customer's response may then be usedby the routing server 124 to route the call or communication to anappropriate contact center resource.

If the communication is to be routed to an agent, the communicationserver 122 interacts with a routing server 124 to find an appropriateagent for processing the interaction. The selection of an appropriateagent for routing an inbound interaction may be based, for example, on arouting strategy employed by the routing server 124, and further basedon information about agent availability, skills, and other routingparameters provided, for example, by a statistics server 132.

In some embodiments, the routing server 124 may query a customerdatabase, which stores information about existing clients, such ascontact information, service level agreement (SLA) requirements, natureof previous customer contacts and actions taken by contact center toresolve any customer issues, and the like. The database may be, forexample, Cassandra or any NoSQL database, and may be stored in the massstorage device 126. The database may also be a SQL database and may bemanaged by any database management system such as, for example, Oracle,IBM DB2, Microsoft SQL server, Microsoft Access, PostgreSQL, MySQL,FoxPro, and SQLite. The routing server 124 may query the customerinformation from the customer database via an ANI or any otherinformation collected by the IMR server 122.

Once an appropriate agent is identified as being available to handle acommunication, a connection may be made between the customer and anagent device 130 a-130 c (collectively referenced as 130) of theidentified agent. Collected information about the customer and/or thecustomer's historical information may also be provided to the agentdevice for aiding the agent in better servicing the communication. Inthis regard, each agent device 130 may include a telephone adapted forregular telephone calls, VoIP calls, and the like. The agent device 130may also include a computer for communicating with one or more serversof the contact center and performing data processing associated withcontact center operations, and for interfacing with customers via voiceand other multimedia communication mechanisms.

The contact center system may also include a multimedia/social mediaserver 154 for engaging in media interactions other than voiceinteractions with the end user devices 108 and/or web servers 120. Themedia interactions may be related, for example, to email, vmail (voicemail through email), chat, voice video, text-messaging, web, socialmedia, co-browsing, and the like. In this regard, the multimedia/socialmedia server 154 may take the form of any IP router conventional in theart with specialized hardware and software for receiving, processing,and forwarding multi-media events.

The web servers 120 may include, for example, social interaction sitehosts for a variety of known social interaction sites to which an enduser may subscribe, such as, for example, Facebook, Twitter, and thelike. In this regard, although in the embodiment of FIG. 1 the webservers 120 are depicted as being part of the contact center system, theweb servers may also be provided by third parties and/or maintainedoutside of the contact center premise. The web servers may also provideweb pages for the enterprise that is being supported by the contactcenter. End users may browse the web pages and get information about theenterprise's products and services. The web pages may also provide amechanism for contacting the contact center, via, for example, web chat,voice call, email, web real time communication (WebRTC), or the like.

According to one exemplary embodiment of the invention, in addition toreal-time interactions, deferrable (also referred to as back-office oroffline) interactions/activities may also be routed to the contactcenter agents. Such deferrable activities may include, for example,responding to emails, responding to letters, attending trainingseminars, or any other activity that does not entail real timecommunication with a customer. In this regard, an interaction (iXn)server 156 interacts with the routing server 124 for selecting anappropriate agent to handle the activity. Once assigned to an agent, anactivity may be pushed to the agent, or may appear in the agent'sworkbin 136 a-136 c (collectively referenced as 136) as a task to becompleted by the agent. The agent's workbin may be implemented via anydata structure conventional in the art, such as, for example, a linkedlist, array, and/or the like. The workbin 136 may be maintained, forexample, in buffer memory of each agent device 130.

According to one exemplary embodiment of the invention, the mass storagedevice(s) 126 may store one or more databases relating to agent data(e.g. agent profiles, schedules, etc.), customer data (e.g. customerprofiles), interaction data (e.g. details of each interaction with acustomer, including reason for the interaction, disposition data, timeon hold, handle time, etc.), and the like. The mass storage device mayalso store a floor plan of the physical contact center, and/or otherphysical characteristics and/or configuration of the contact center.According to one embodiment, some of the data (e.g. customer profiledata) may be maintained in a customer relations management (CRM)database hosted in the mass storage device 126 or elsewhere. The massstorage device may take form of a hard disk or disk array as isconventional in the art.

According to some embodiments, the contact center system may include auniversal contact server (UCS) 127, configured to retrieve informationstored in the CRM database and direct information to be stored in theCRM database. The UCS 127 may also be configured to facilitatemaintaining a history of customers' preferences and interaction history,and to capture and store data regarding comments from agents, customercommunication history, and the like.

The contact center system may also include a reporting server 134configured to generate reports from data aggregated by the statisticsserver 132. Such reports may include near real-time reports orhistorical reports concerning the state of resources, such as, forexample, average waiting time, abandonment rate, agent occupancy, andthe like. The reports may be generated automatically or in response tospecific requests from a requestor (e.g. agent/administrator, contactcenter application, and/or the like).

The contact center system may also include an analytics server 158 thatautomatically processes interactions between the end users 106 andagents of the contact center to generate interaction data regarding theinteractions, including the reason for the interaction, topics discussedduring the interaction, customer sentiment, agent sentiment, interactionescalation rates, etc. The analytics server 158 may include a voiceanalytics component configured to recognize speech in voice or audiointeractions between the end users 106 and the agents of the contactcenter in order to detect events in the voice or audio interactions. Theanalytics data generated by the analytics server 158 may be provided tothe statistics server 132 and may be stored on the mass storage device126 or elsewhere.

The contact center system may also include a workforce management server160 (or workforce management system or employee engagement system orworkforce optimization system) to generate work schedules for agents inaccordance with expected interaction or call volumes and in accordancewith agent skills (e.g., language or technical skills), schedulinginformation (e.g., agent starting, ending, and break times), or otherinformation that may be useful to improve the efficacy of routing tasks.For example, in a contact center that operates 24 hours per day, areduced call volume may be expected during the nighttime hours than thedaytime hours, and therefore the staff size may be smaller. As anotherexample, when a company releases a new product, technical supportstaffing may be increased to handle expected increased traffic fromcustomers requesting help in setting up the new products.

The workforce management server 160 may also provide a user interface toallow a human supervisor or administrator to visualize and to generateagent work schedules in accordance with the agent data and expecteddemand. The workforce management server 160 may also provide a userinterface for agents to, for example, submit preferences such as shiftscheduling preferences, submit requests for time off (e.g., sick timeand vacation time), and view assigned work schedules. The user interfacemay be, for example, web server software that provides a web interfaceto the workforce management server 160, an application programminginterface (API) that allows access from a client program, or other knowntechniques for providing user interfaces to computer systems.

The various servers of FIG. 1 may each include one or more processorsexecuting computer program instructions and interacting with othersystem components for performing the various functionalities describedherein. The computer program instructions are stored in a memoryimplemented using a standard memory device, such as, for example, arandom access memory (RAM). The computer program instructions may alsobe stored in other non-transitory computer readable media such as, forexample, a CD-ROM, flash drive, or the like. Also, although thefunctionality of each of the servers is described as being provided bythe particular server, a person of skill in the art should recognizethat the functionality of various servers may be combined or integratedinto a single server, or the functionality of a particular server may bedistributed across one or more other servers without departing from thescope of the embodiments of the present invention.

In the various embodiments, the terms “interaction” and “communication”are used interchangeably, and generally refer to any real-time andnon-real time interaction that uses any communication channel including,without limitation telephony calls (PSTN or VoIP calls), emails, vmails(voice mail through email), video, chat, screen-sharing, text messages,co-browsing (e.g., the customer and agent viewing the same web pagestogether, where an action taken by either the customer or the agentupdates the views seen by both the customer and the agent), social mediamessages, web real-time communication (e.g., WebRTC calls), and thelike.

Customizing a Language Model

FIG. 2 is a schematic diagram of a speech recognition engine 44according to one embodiment of the present invention. The speechrecognition engine may also be referred to as an automatic speechrecognition (ASR) system, which includes large vocabulary continuousspeech recognition (LVCSR) systems, although embodiments of the presentinvention are not limited thereto. Generally, a speech recognitionsystem 44 may include a training module 44A and a decoder module 44B.The training module 44A may include a language model training module44AA and an acoustic training module 44AB. The language model trainingmodule 44AA may generate a language model (LM) and the acoustic trainingmodule 44AB may generate an acoustic model (AM) (e.g., a deep neuralnetwork).

The decoder module 44B uses the language model and the acoustic model toprocess input audio and to generate a speech recognition output (e.g., atranscription or text) of the speech that is recognized from the inputaudio. The decoder module 44B may also use a grapheme to phoneme model(G2P). In particular, the input audio is supplied to the acousticmodels, which generate likely phonemes. The decoder 44B combines theoutput of the acoustic models and the output of the language model, andidentifies the likely words spoken in the audio based on theprobabilities of various phrases and the phonemes detected in the audio.

In more detail, the acoustic model maps sounds to features (e.g.,numerical vectors such as a floating point vector) to identify likelyphonemes found within the audio. The identification of the phonemes maybe based on “tri-phones” (a sequence of three phonemes) where eachphonetic unit includes the current phoneme, the previous phoneme, andnext phoneme. The acoustic model is probabilistically mapping acousticfeature vectors into the space of all possible tri-phones in a givenlanguage. Using a tri-phone improves performance because some phonemesare more likely to occur before or after certain other phonemes. Theacoustic model may also take into account data from the global trainingmaterial (e.g., the likelihoods of particular phonemes in particularlanguages, dialects, or accents), where the global training material mayinclude data associated with a particular language (e.g., English,Chinese, or French) and/or a particular region or accent (e.g., AmericanEnglish or Australian English). In one embodiment, the acoustic modelsare deep neural networks (e.g., neural networks having more than threelayers). The acoustic model also takes as input a lexicon, whichcontains phonetic spellings of words (e.g., multiple phonetic spellings)that may be spoken in the audio. As such, the acoustic training modulegenerates acoustic models using an input lexicon and a collection ofaudio (e.g., audio collected from prior calls) and correspondingtranscription of that audio.

The language model provides information regarding the probability that agiven input phrase appears in the context of the input data used totrain the language model. Language modeling is generally mediumsensitive. For example, a language model trained on transcripts ofspeech interactions will generally be different from a language modeltrained on newspaper articles. While it may be relatively easy to mixdata from two different sources in the same medium (e.g., transcripts ofcontact center interactions and transcripts of television programs),mixing two different sources of data from different media (e.g., speechrecognition transcripts and emails) requires more careful selection, asdescribed in more detail below. The performance of an automatic speechrecognition system is sensitive to different language models in that theperformance of the speech recognition can depend on how well thelanguage model matches the actual probabilities of various phrasesappearing in the input audio. For example, if the language model wastrained using general English language training material, the phrases“How are you?” and “thank you” may be fairly common, whereas “ni haoma?” (e.g., “How are you?” in Chinese) may be much less common. Asanother example, a language model generated based on the context ofinteractions between customers and agents of a contact center wouldestimate that the sequence of words “will you please verify youraddress” to have a higher probability than the sequence of words: “mydog likes the park by the river.”

As such, when a language model is customized to the context in which itwill be operating, it can provide information for differentiatingbetween multiple phrases that sound similar (e.g., utterances thatproduce similar results from the acoustic model) based on informationabout the likelihood of various phrases, as encoded in the languagemodel.

In a speech analytics system 45, there may be at least two differentcorpora of training material. One of these corpora may be referred toherein as “global training material” (or “generic training material” or“general training material”) and may include words (e.g., millions ofwords) collected from a large number of sources (e.g., transcripts ofinteractions from contact centers of a large number of differentorganizations), and this “global training material” may be useful forcustomizing the language model for a particular human language such asEnglish, Spanish, or Mandarin Chinese.

Another corpus of these corpora may contain a set of topics and phrasesdefined specifically for one particular organization, and may bereferred to herein as “organization specific” data (or may be referredto as “organization phrases,” “customized phrases,” “customer phrases,”or “customer training material,” where the organization is the“customer”). These organization specific training data includeparticular topics and phrases that may be of higher value to theparticular organization (e.g., for a company that sells compilersoftware, these may be topics and phrases relating to compilers,software, programming, etc.), and can therefore reduce the ambiguitybetween two valid phrases that sound alike, but where one transcriptionmay be more likely than another in the context of the organization.

For example, in the context of interactions between customers of a petsupply retailer and its contact center, words or phrases such as:“reptile,” “live mice,” “my dog won't eat this food, can I return it?”,and “can you recommend a brand of cat litter?” might appear more oftenthan would be expected in a typical contact center (or average of allcontact centers or the global in-medium training material) and thesephrases would therefore not be strongly reflected in the globalin-medium training material.

Similarly, in the context of interactions between customers of asoftware compiler company and the technical support staff of its contactcenter, words or phrases such as: “compiler,” “linker,” “regex,”“segfault,” and “how do I cross-compile my project to a differentarchitecture?” might appear more often than would be expected in atypical contact center (or average of all contact centers) and thesephrases would therefore not be strongly reflected in the globalin-medium training material.

These language models may be trained on language data from the samemedium that they are intended to be used with. For example, a languagemodel for predicting the likelihood of sequences of words spoken in acontact center interaction will be trained on transcripts of voiceinteractions of the contact center (sometimes referred to as “in-domain”or “in-medium” data), as opposed to other types of language data (e.g.,non-voice data) such as transcripts of typed text chats, emailexchanges, forum posts, website data, and the like (sometimes referredto as “out-of-domain” or “out-of-medium” data). However, in somecircumstances, there may not be sufficient existing data to perform thistraining, such as in the case of a new contact center that has noexisting set of recorded interactions or very few recorded interactions.In some circumstances, the recorded interactions may be available, butit may be expensive to generate the transcripts (because the transcriptsare typically manually generated by a human).

The terms “in-domain” and “out-of-domain” or “in-medium” and“out-of-medium” do not constrain whether the data is “organizationspecific,” and therefore there may be “organization specific in-mediumdata” (e.g., recordings of voice interactions between agents of theorganization and its customers, and transcripts thereof), “organizationspecific out-of-medium data” (e.g., non-voice interaction data of theorganization), “non-organization specific in-medium data” (e.g., voiceinteraction data from other organizations), and “non-organizationspecific out-of-medium data (e.g., non-voice interaction data from otherorganizations).

In one approach, the speech recognition engine uses only the globaltraining material, which may include transcripts from a variety ofdifferent contact centers, to produce a language model that is nottailored to the context of the organization because it may lack theorganization phrases. This general language model may then be used torecognize speech captured from voice interactions with between acustomer (e.g., a third party contacting the organization) and an agentto generate speech recognizer output (or “LVCSR output”).

However, a speech recognition system 44 using this general languagemodel may exhibit high word error rates (WER) because contact centerspeech transcript data from one contact center may have low relevance tothe context that the speech recognition system is being trained for. Forexample, a contact center for a pet supply retailer may frequentlyencounter the phrase “I would like a new leash,” but a general languagemodel trained with transcripts from, for example, a large apartmentrental company may assign a higher likelihood to the phrase: “I wouldlike a new lease.”

Therefore, one challenge of an automatic speech recognition system is touse a language model that is specific enough for the relevant context,yet generic enough to recognize new sentences and words by training alanguage model with a large volume of reliable data from the samecontext, such as business or business vertical, as the intended contextof the language model. In this way, the speech recognition system canprovide better results, such as in the form of lower word error rates.However, generating the corpora of training material, whether global ororganization-specific, is generally a long, manual, and expensiveprocess, in which human auditors transcribe real audio recordings (e.g.,recordings of voice interactions between customers and agents of anorganization) into text.

As such, aspects of embodiments of the present invention are directed tothe customization of a generic language model for a target context usingdata that were manually generated for the target context, existingspeech recognition transcripts from the target context (if any), andout-of-medium data from the target context (e.g., data other than speechrecognition transcripts from the context of the organization). Thetarget context may be, for example, a target organization using thelanguage model to recognize speech in its contact center. The datagenerated for the target context may be referred to as the “program” andmay include representations of data that are expected to be found inthose contexts. For example, an organization may expect phrases relatingto specific products in sales and support interactions, whereas theorganization may expect phrases relating to account verification,payment, and balances in billing interactions. This “program”information may be used to select a speech recognition system to deployin accordance with the context of the interaction (e.g., billing versussales versus support), which each speech recognition system may have alanguage model tailored for its context. The out-of-medium data from thetarget context may include, for example, text chat transcripts betweencustomers of the organization and agents, emails between customers andagents, and text information obtained from the organization websiteand/or sales and support materials.

Generally, embodiments of the present invention are directed to systemsand methods for extracting useful training data from the out-of-mediumdata of the target organization. In addition, embodiments of the presentinvention are also directed to computing the similarity of the genericor general contact center data to the target context, and using thesecomputed similarities to generate a new language model for the targetcontext. In some embodiments, organization-specific data from othersources within the target context, such as text chats and emails withthe organization's agents, the organization's website, and theorganization's customer forums, may be used to assist in rating thegeneric or general contact center data.

Aspects of embodiments of the present invention include the creation ofa tailor made language model using “in-medium” data that are ranked andweighted in accordance with its similarity to the target context; usingout-of-medium data to assist in ranking the in-medium data andcalculating weights; customization of a language model using the“program” of expected topics or topics of interest to the organization;and mixing and weighting several data sets (that may have very differentsizes) into one language model. As such, embodiments of the presentinvention allow customization of a language at “time zero” (e.g.,because the organization has not been operating a contact center, beforeany company specific in-medium data is collected, or before theorganization has collected enough in-medium training data to beworthwhile to include the data). Embodiments of the present inventionalso enable on-going customization of the language model as time passesand new vocabulary is introduced to the system (e.g., in new lines ofproducts dealing with new concepts and new ideas, reorganization andrenaming of product lines, new types of problems, and the like). Sourcesof information for the on-going customization of the language modelinclude the results of the speech recognition output of the voiceinteractions, emails, text chats, customer forums, frequently askedquestions and answers, and the “program” (e.g., topics of interest tothe organization and specified in the voice analytics system 45).

FIG. 3 is a flowchart of a method for generating a customized languagemodel according to one embodiment of the present invention. Referring toFIG. 3, the method includes: extracting organization specific trainingdata from initial organization-specific training material in operation200; computing a similarity between the organization specific trainingdata and the in-medium training data 300; sorting the in-medium trainingdata according to the computed similarities 400; partitioning thetraining data into partitions 500; weighting the partitions based oninterpolation parameters 600, and training a new language model inaccordance with weighted training data 700.

In order to customize the language model, in operation 200, the LMtraining module 44AA obtains organization-specific data. For ongoingcustomization of the language model, the text output of the speechrecognition module 44 can be supplied as organization-specific trainingdata. However, if no such training data are available (e.g., in the“time-zero” situation), then other initial organization specifictraining material can be used, such as the “program” and existingout-of-medium data (e.g., text chats, emails, text scraped from theorganization's website, forum posts, and the like).

The various out-of-medium data and generic data may differ from thein-medium organization-specific data in a number of ways, includingmodality (e.g., voice versus text chat), context (e.g., contact centerversus television), business vertical (e.g., travel versus finance), andever type of communication (e.g., phone conversation versus voice mail).Nevertheless, in many cases, these various out-of-medium and genericdata contain relevant material that can be carefully selected.

As such, some aspects of embodiments of the present invention relate toselecting relevant material from out-of-medium and generic data. In someembodiments, relevant material may be selected from the out-of-mediumand generic data on a sentence level, thereby creating customizedtraining data at a lower expense and higher level of automation, wherethe customized training data can be used to generate a language modelfor higher performance speech recognition in the target context. Aspectsof embodiments of the present invention are also directed to theautomatic update of the language model in accordance with the selectionof relevant data from newly added sources of out-of-medium and genericmaterial.

According to some embodiments of the present invention, out-of-mediumdata can be combined with a generic language model trained usingin-medium data or combined with a customized language model to generatea customized language model. In addition, aspects of embodiments of thepresent invention are directed to updating a language model inaccordance with additional or new out-of-medium material.

FIG. 4A is a flowchart of a method 200 for extractingorganization-specific data from a plurality of sources according to oneembodiment of the present invention. The method for extracting relevantdata includes converting the data to in-medium compatible data (e.g.,speech transcription compatible data) in operation 210, and selectingdata from the cleaned data in operation 270.

Operation 210 may include a plurality of operations to convert theout-of-medium data into data that are compatible with the in-medium databy making the data more closely resemble in-medium data. In someembodiments of the present invention, “speech transcription compatibledata” refers to text data that contain phrases that are consistent withspeech transcription text, such as types of phrases, spelling,abbreviation, pronunciation of acronyms, and the like, and where thedistribution or frequency of these phrases in the data are consistentwith their distribution in transcriptions of speech data. This processmay be referred to as “cleaning” the data. FIG. 4B is a flowchart of amethod for cleaning out-of-medium data according to one embodiment ofthe present invention. The cleaning of the data may include: filteringthe words in accordance with the confidences in the words (e.g., thedecoder 44B assigns confidences to the words in the transcripts, wherethe confidence represents the likelihood that the word was correctlydecoded from audio to text) and extracting sentences according to partof speech sequences (e.g., removing sentences that are syntacticallyincorrect and keeping sentences that are syntactically correct, asidentified by the part of speech sequences) so that later stages uselegal sentences.

In operation 212, the LM training module 44AA removes metadata, such asremoving headers (and footers) of the documents which are not part ofthe content of the conversation, interaction, or document. For example,in the case of the transcript of a text chat or in forum posts,usernames and timestamps may be automatically removed during the headerand footer removal operation. As another example, in an email, theheaders of the email, such as the “to” and “from” addresses, routinginformation, timestamps, and other data may be removed so that only thebody (and, in some embodiments, the subject line) of the email is leftfor further processing.

In operation 214, the LM training module 44 performs sentence splittingon the data. In this operation, the LM training module 44AA splits thetext of the out-of-medium data into sentences. This may done, forexample, based on punctuation, time gaps, a sentence splitting algorithmsuch as the Sentence Splitter module of FreeLing (see, e.g., Lluís Padróand Evgeny Stanilovsky. FreeLing 3.0: Towards Wider Multilinguality.Proceedings of the Language Resources and Evaluation Conference (LREC2012) ELRA. Istanbul, Turkey. May, 2012.), and the like.

In some embodiments, in operation 216, the LM training module 44AAconverts text that was originally written or typed into a spoken form.For example, symbols such as dates in a numeric format (e.g., 5/24/2016)or digits (e.g., 13) are converted into the words that are typicallyused when spoken (e.g., respectively, “May 24, 2016” or “thirteen”).Similarly, abbreviations are converted to the spoken form, e.g. from“USD” to “dollars,” from “IMO” to “in my opinion,” and the like. Inaddition, in operation 218, common spelling mistakes that are known inadvance may also be converted to the correct form (e.g., from “teh” to“the,” from “u” to “you,” “tho” to “although,” and the like).

In some embodiments, the cleaning of the data in operation 210 alsoremoving sentences that contain unrecognized words that are determinedto be misspellings rather than legitimate new words that are not in thevocabulary V in operation 230. As such, aspects of embodiments of thepresent invention relate to further customizing the language model basedon changes to the vocabulary V used in the context in which the languagemodel operates by detecting “new” words and adding them to the languagemodel. On the other hand, new words should be distinguished from commonspelling mistakes that are not really new words. The erroneous additionof misspelled words to a language model can cause an undesirable effectin which the speech recognition system 44 outputs these misspelled wordsin a transcript. Therefore, aspects embodiments of the present inventionrelate to a method for distinguishing between new words and spellingmistakes, in which new words are added to the cleaned training data forthe language model, and spelling mistakes are removed.

The LM training module 44AA may enlarge the dictionary or vocabulary Vbased on known words. These words may include words from the “program”created by human analysts of the analytics data. These words may alsoinclude words from existing spelling dictionaries such as Hunspell.

FIG. 5 is a flowchart of a method 230 according to one embodiment of thepresent invention for filtering out sentences from the out-of-mediumtraining data to remove sentences with spelling errors, while leavingsentences containing new words.

In operation 232, the LM training module 44AA selects a next word in thevocabulary V to process. In operation 234, it identifies sentences ofthe out-of-medium training data that contain the current word. Inoperation 236, the LM training module 44AA determines whether the countC of the word w (the number of times the word appears in theout-of-medium training material C(w)) is less than a threshold c₁. Thisparameter and this value may be respectively denoted herein asMinigCount=c₁. If the count of the word is greater than the thresholdc₁, then, in operation 238, the identified sentences containing the wordare added to an output set of out-of-vocabulary training data. In eithercase, the flow proceeds to operation 240, in which the LM trainingmodule 44AA determines whether there are more words in the vocabulary Vto process. If so, then the flow returns to operation 232 to select thenext word. If not, then the flow proceeds to operation 242.

If there are no more in-vocabulary words to process, then in operation242, the LM training module 44AA identifies a collection ofout-of-vocabulary (OOV) words {w} appearing in the out-of-mediumtraining data. In operation 244, the LM training module 44AA identifiesa next OOV word w from the set of OOV words {w}, and computes a countC(w) of the OOV word w in operation 248. If the count C(w) does notexceed a second threshold c₂ (which may be referred to as Min2gCount),where c₂>c₁ (e.g., c₂=2c₁), then the flow proceeds to operation 256. Ifthe count of the word is higher than the threshold, then in operation250, OOV word w is provided to a spell checking program (e.g.,Hunspell), and the spell checking program generates a collection of oneor more spelling suggestions {s} as to potential correct spellings ofthe OOV word w. In operation 252, the LM training module 44AA evaluatesthe inequality:P(w|h)>P(s|h)for every suggestion s in the collection {s}, where h is a history(e.g., words immediately preceding the OOV word w). The probabilitiesP(w|h) and P(s|h) can be computed from the counting the respectiveoccurrences of word w and suggestion s preceded by history h in theout-of-medium training data (C(h,w) and C(h,s), respectively). In otherwords, the LM training module 44AA computes, for each s in {s}, whetherit is more probable to find w or s given a preceding set of words h.

Because the count of the history C(h) is constant, the above inequalityreduces to the condition:C(w,h ₁ , . . . ,h _(n))>C(s,h ₁ , . . . ,h _(n))

In some embodiments of the present invention, the words following theOOV word w may also be used to compute the probabilities. Given that hrepresents a history (a sequence of words that come before the OOV wordw in the sentence), f represents a sequence of future words (a sequenceof words that come after the OOV word w in the sentence). Therefore, ina similar way to the above, the criteria would be:P(w|h,f)>P(s|h,f)

While these probabilities could be calculated based on countingoccurrences in the out-of-medium data, the computation can besimplified:

${P\left( {{w❘h},f} \right)} = \frac{P\left( {w,{f❘h}} \right)}{P\left( {f❘h} \right)}$

The constant denominator can be ignored because the goal is to find amaximum (“argmaxing”) on w and the collection of suggestions {s}, so itis sufficient to compute P(w,f|h) as:

${P\left( {w,{f❘h}} \right)} = \frac{C\left( {h,w,f} \right)}{C(h)}$In other words, the LM training module 44AA can compute the probabilityP(w,f|h) (the probability of encountering word w followed by futurewords f, given previous words h leading up to word w) by counting thenumber of times the sequence (h,w,f) appears in the out-of-mediumtraining data and by dividing by the total number of times h appears inthe out-of-medium training data.

If the inequality is satisfied for all suggestions s, then the LMtraining module 44AA considers the OOV word w to likely be a new wordrather than a misspelling. In other words, if this OOV word appearssurrounded by a history and future words more often than any of thesuggested corrected spellings, then it is considered likely to be a newword, and the identified sentences are added to the output set inoperation 254. In addition, the OOV word may be added to the lexicon, oradded to a lexicon of new words. However, if the OOV word fails eitherof the inequalities, then it is considered to be a misspelled word andthe flow proceeds to operation 256.

In operation 256, the LM training module 44AA determines whether thereare more words in the set of OOV words. If so, then the flow returns tooperation 244, in which a next OOV word is selected. If there are nomore words, then the process of removing sentences that contain spellingmistakes rather than new out-of-vocabulary words is complete, and theprocess can continue with further cleaning in operation 210.

In some embodiments, some limited human involvement can be included inoperation 260 to perform manual spelling corrections for some words.

The set of sentences remaining in the cleaned out-of-medium trainingdata after the spelling corrections, but before the next operationbelow, will be referred to as a set of sentences G.

In some embodiments, in operation 210, the LM training module 44AAremoves sentences in which a large number of the words areout-of-vocabulary (OOV). In one embodiment, if more than a thresholdnumber of words in a sentence are not in a lexicon (e.g., if more thantwo of the words in the sentence are not in the lexicon), then theentire sentence is excluded from further analysis. This cleaningoperation may be used to remove sentences that are not relevant because,for example, they are written in a different language (e.g., a sentencewritten in Spanish where the target context is English).

In operation 210, the LM training module 44AA may also remove duplicatesentences.

The set of sentences remaining in the cleaned out-of-medium trainingdata after removing the duplicate sentences will be referred to as a setG′ of unique sentences.

In operation 270, the LM training module 44AA selects data from thecleaned out-of-medium training material selected in operation 210.

According to one embodiment of the present invention, the LM trainingmodule 44AA selects data from the out-of-medium training material usingan in-medium (or in-domain) out-of-medium (or out-of-domain) crossentropy difference technique. In this embodiment, the two sets ofsentences G and G′ are supplied as input along with the global in-mediumtraining data, and parameters MinLMScore, MaxLMScore, and MinDelta. FIG.6A is a flowchart of a method 270 for selecting data from theout-of-medium training material using an in-medium/out-of-medium crossentropy difference technique according to one embodiment of the presentinvention.

In operation 272, the LM training module 44AA replaces low occurrenceunigrams in the global in-medium training material with a placeholdersymbol (e.g., <UNK>), and, in operation 274, computes an in-mediumlanguage model IDLM from the resulting global in-medium trainingmaterial.

In operation 276, the LM training module 44AA generates an out-of-mediumlanguage model OODLM using the set of cleaned out-of-domain sentences G.

The LM training module 44AA iterates over the sentences s in G′ anddetermines whether to add the sentence s to the result set Q. Inoperation 278, the LM training module 44AA selects a next sentence sfrom G′ to evaluate (e.g., randomly selects a next sentence s). Inoperation 282, the LM training module 44AA supplies the sentence s tothe in-medium language model IDLM to compute an in-medium score IDScr,and, in operation 284, supplies the sentence s to the out-of-mediumlanguage model to compute an out-of-medium score OODScr. The LM trainingmodule then normalizes the scores by the length of the sentence toproduce a normalized IDScr′ and a normalized OODScr′ in operation 286and 288, respectively.

In operation 290, the LM training module 44AA determines whether theIDScr′ is between MinLMScore and MaxLMScore(MinLMScore≤IDScr′≤MaxLMScore) and, in operation 292, whetherIDScr′−OODScr′≥MinDelta. If either of the conditions fails, then theflow proceeds to operation 296. However, if both conditions of 290 and292 are satisfied, then in operation 294, the LM training module 44AAadds s to a result set Q (with the original words of s, rather than withunigrams replaced with <UNK>), along with its score.

In operation 296, the LM training module 44AA determines whether thereare more sentences in G′. If so, then the process returns to operation278 to select the next sentence s from G′. If not, then the processproceeds to output the set Q in operation 298.

FIG. 6B is a flowchart of a method 800 for computing the weights ofout-of-medium sentences according to one embodiment of the presentinvention. In operation 600, the LM training module 44AA counts thenumber of sentences in G′ as a value c. In operation 804, the LMtraining module 44AA iterates over the sentences s to converts thenormalized score IDScr′ of each sentence s to a probability scale, wherethe probability of a sentence P(s) is given by:P(s)=e ^(IDScr′)

In operation 806, the LM training module 44AA normalizes the computedprobabilities P so that they sum to 1. For example, the LM trainingmodule 44AA may compute a value k such that:

${\sum\limits_{s}^{G^{\prime}}{P(s)}} = k$and may then normalize the probabilities P(s) by scaling eachprobability P(s) by 1/k.

In operation 808, the LM training module 44AA sentences s in Q to setthe count Count(s) of each sentence s in the resulting training data tobe:Count(s)=Max(1,floor(P(s)*c))

As such, the out-of-medium data can be selected and scaled for use withthe global in-medium training data to produce scaled out-of-domaintraining data for training a customized language model.

According to another embodiment of the present invention, the LMtraining module 44AA selects data for the statistical language model(SLM) using a cross entropy minimization technique. FIG. 7A is aflowchart of a method 810 for selecting data from out-of-medium trainingdata using cross entropy minimization according to one embodiment of thepresent invention. Generally, the selection method in embodiments of thepresent invention selects sentences that, in some sense, do not causelarge changes to the in-domain language model distribution. Morespecifically, in some embodiments of the present invention, the LMtraining module 44AA only sentences that reduce the Kullback-Leibler(KL) divergence between the trained language model and the reference(global) language model.

According to one embodiment of the present invention, a referencetraining set P includes the global in-medium training data and selects anew set Q that includes the selected sentences s. In operation 811, theLM training module 44AA receives a set of cleaned out-of-mediumsentences G′. For each new sentence s in the out-of-medium training dataG′, the LM training module 44AA adds the sentence s to Q in operation815 if, in operation 813, the sentence s satisfies the condition:D _(KL)(P∥Q∪{s})<D _(KL)(P∥Q)whereD _(KL)(P∥Q)=H(P,Q)−H(P)where H(P,Q) is a cross-entropy between P and Q and H(P) is the entropyof P. In some embodiments, the sentences s are added from G′ in randomorder. In operation 817, the LM training module determines if there aremore sentences s in G′ to be evaluated. If so, then in the flow returnsto operation 811 to select the next sentence. If there are no moresentences, then in operation 819, the resulting set Q is output to becombined with the global in-domain training material.

However, calculating the Kullback-Leibler (KL) divergence D_(KL) foreach sentence s in operation 813 can be computationally expensive. Assuch, according to one embodiment of the present invention, a simplifiedcalculation can be used to arrive at similar results. Because Pcorresponds to the global in-medium training data, which is constantthrough the process of evaluating the sentences s, minimizingD_(KL)(P∥Q) is equivalent to minimizing H(P,Q). This, in turn, isequivalent to minimizing the perplexity PP(P,Q) of Q given a test P,where P is the empirical distribution in the set P (in other words,P(x)=n/m where n is the number of times that x appears in the test set Pof size N. Therefore, the cross-entropy term of the empiricaldistribution can be expressed as:

${H\left( {P,Q} \right)} = {{- \frac{1}{N}}{\sum\limits_{x_{i} \in P}{\ln\;{Q\left( x_{i} \right)}}}}$by ignoring the constant term

${- \frac{1}{N}},$minimizing H(P,Q) is equivalent to maximizing:

$\hat{H} = {\sum\limits_{x_{i} \in P}{\ln\;{Q\left( x_{i} \right)}}}$By the chain rule:

${\ln\;{Q\left( x_{i} \right)}} = {{\ln\;{Q(i)}} = {{\sum\;{\ln\frac{tri}{bi}}} + {\ln\frac{bi}{un}} + {\ln\frac{un}{M}}}}$where M is the number of sentences in Q, un is the count of thesentence-starting leftmost unigram from x_(i), in Q, bi is the count ofthe leftmost bigram from x_(i), in Q, and tri is the count of thecurrent trigram (in the summary) from x_(i), in Q. When M is large,ln(M)≈ln(M+1), and therefore, in some embodiments, substantially thesame results can be obtained without updating the number M for every newsentence s. For example, the number M can be updated for every time thesize of Q increases by approximately M (e.g., each time Q doubles insize, as approximated by the number of sentences s that are considered).

Substituting the expression for ln Q(x_(i)) into the above sum resultsin:

$\hat{H} = {{\sum\limits_{u \in {tri}}{{C(u)}\ln\frac{u}{{bi}(u)}}} + {\sum\limits_{v \in {bi}}{{C(v)}\ln\frac{v}{{un}(v)}}} + {\sum\limits_{w \in {un}}{{C(w)}\ln\frac{w}{M}}}}$where C is the count of the n-gram (e.g., trigram, bigram, or unigram)in the set.

Given the above simplification of the evaluation of the cross-entropy Ĥof adding a sentence x_(i) to the resulting set of sentences Q, indifferent embodiments of the present invention, the sentences aremodeled in different ways for evaluation.

In one embodiment of the present invention, a sentence s is modeled asan n-gram. FIG. 7B is a flowchart of a method 820 for evaluating asentence s modeled as an n-gram to update the numerators in the aboveexpression for Ĥ

$\left( {{e.g.},{{the}\mspace{14mu}{numerators}\mspace{14mu}{of}\mspace{14mu}{C(u)}\ln\frac{u}{{bi}(u)}\mspace{14mu}{or}\mspace{14mu}{C(v)}\;\ln\frac{v}{{un}(v)}\mspace{14mu}{or}\mspace{14mu}{C(w)}\ln\frac{w}{M}}} \right)$in accordance with one embodiment of the present invention. In operation821, the LM training module 44AA divides the sentence s into a pluralityof unigrams {w}, to update the relevant bins (each bin referring to anelement in the summary Ĥ), for each unigram w in s (so that only thebins corresponding to unigrams w from s are updated, not all elements inthe summary Ĥ). In operation 823, the LM training module 44AA selects anext unigram w from the plurality of unigrams {w}. In operation 825, thecross-entropy Ĥ is updated:

$\left. \hat{H}\leftarrow{\hat{H} - {{C(w)}\ln\frac{w}{N}} + {{C(w)}\ln\frac{w + 1}{M^{\prime}}}} \right. = {\hat{H} + {{C(w)}\left\lbrack {{\ln\frac{w + 1}{w}} + {\ln\frac{M}{M + 1}}} \right\rbrack}}$where M′=M+1. When M is large, M′≈M, so ln(M/(M+1))≈0, and therefore theupdate of cross-entropy Ĥ can be expressed as:

$\hat{H}+={{C(w)}\ln\frac{w + 1}{w}}$where “+=” refers to “augmented assignment” (x+=a is defined as x←x+a).

In operation 829, the LM training module 44AA divides the sentence sinto a plurality of bigrams {v} (two consecutive words) and selects thenext bigram v in operation 831. For the current bigram v, in operation831, the LM training module 44AA updates the entropy Ĥ as follows:

$\left. \hat{H}\leftarrow{\hat{H} - {{C(v)}\ln\frac{v}{{un}(v)}} + {{C(v)}\ln\frac{v + 1}{{un}^{\prime}(v)}}} \right.$where un′(v) is the new count of un(v) after the previous step. This isequivalent to:

$\hat{H}+={{C(v)}\left\lbrack {{\ln\frac{v + 1}{v}} + {\ln\frac{{un}(v)}{{un}^{\prime}(v)}}} \right\rbrack}$

In operation 835, the LM training module determines whether there aremore bigrams v in the plurality of bigrams {v}. If so, then the processreturns to operation 831, in which the next bigram v is selected.

Similarly, in operation 837, the LM training module 44AA divides thesentence s into a plurality of trigrams {u} (three consecutive words)and selects a next trigram u in operation 839. In operation 841, the LMtraining module updates the entropy Ĥ based on the trigram u:

$\hat{H}+={{C(u)}\left\lbrack {{\ln\frac{u + 1}{u}} + {\ln\frac{{bi}(u)}{{bi}^{\prime}(u)}}} \right\rbrack}$where bi′(u) is the new count of bi(u) after the previous step.

In operation 843, the LM training module determines whether there aremore trigrams u in the plurality of trigrams {u} to process. If so, thenthe flow returns to operation 839 to select the next trigram u. If not,then the process is complete, and the computed cross entropy Ĥ of addingthe current sentence s is returned.

In one embodiment, the LM training module 44AA updates the denominator.FIG. 7C is a flowchart illustrating a method 850 for updating thedenominators in the above expression for Ĥ

$\left( {{e.g.},\;{{the}\mspace{14mu}{denominators}\mspace{14mu}{of}\mspace{14mu}{C(u)}\ln\frac{u}{{bi}(u)}\mspace{14mu}{or}\mspace{14mu}{C(v)}\ln\frac{v}{{un}(v)}\mspace{14mu}{or}\mspace{14mu}{C(w)}\ln\frac{w}{M}}} \right)$in accordance with one embodiment of the present invention. In operation851, the LM training module 44AA identifies bigrams v in Q prefixed withunigram w in s (∀v∈Q prefixed with unigram w∈s). In operation 853′ theLM training module 44AA selects one of the bigrams v from among theidentified bigrams and, in operation 855, updates the entropy Ĥ:

$\left. \hat{H}\leftarrow{\hat{H} - {{C(v)}\ln\frac{v}{{un}(v)}} + {{C(v)}\ln\frac{v + 1}{{un}^{\prime}(v)}}} \right. = {{\hat{H} + {{C(v)}\ln\frac{{un}(v)}{{un}^{\prime}(v)}}} = {\hat{H} + {{C(v)}\ln\frac{w}{w + 1}}}}$

In operation 857, the LM training module 44AA determines whether thereare more bigrams v in the plurality of bigrams {v} to be processed. Ifso, then the process returns to operation 853 to select the next bigramv. If not, then the process continues to operation 859.

In operation 859, the LM training module 44AA identifies every trigram uin Q that is prefixed with bigram v (∀u∈Q prefixed with v∈Q) to generatetrigrams {u}. In operation 861, the LM training module selects a nexttrigram u from {u}, and in operation 863, updates the entropy Ĥ:

$\left. \hat{H}\leftarrow{\hat{H} - {{C(u)}\ln\frac{u}{{bi}(u)}} + {{C(u)}\ln\frac{u}{{bi}^{\prime}(u)}}} \right. = {\hat{H} + {{C(u)}\ln\frac{v}{v + 1}}}$

In operation 865, the LM training module 44AA determines whether thereare more trigrams u in the plurality of trigrams {u} to be processed. Ifso, then the process returns to operation 861 to select the next trigramu. If not, then the process is complete and the resulting entropy Ĥ isreturned.

Regarding out-of-vocabulary words from P (unigrams w), in order to avoidcalculating entropies of −∞, and in order to model low values forout-of-vocabulary words (that will increase Ĥ when they are added to thevocabulary to become in-vocabulary words), all low-occurrence unigramsare replaced with a common token in Q (e.g., “<UNK>”). In one embodimentof the present invention, unigrams w having a count less than athreshold level c are replaced with the token “<UNK>.” In addition, whencomputing the entropy Ĥ, words (unigrams w) occurring less than c timesare replaced with the token “<UNK>.”

The unknown tokens may be periodically recomputed, but embodiments ofthe present invention do not necessarily require that these berecomputed for every sentence s that is added.

In addition, embodiments of the present invention may apply a back-offmodel to avoid problems of out-of-vocabulary n-grams of higher order,such as bigrams and trigrams, that nullify the whole term. One exampleof a back-off model is described in Katz, Slava M. Estimation ofProbabilities from Sparse Data for the Language Model Component of aSpeech Recognizer. IEEE Trans. on Acoustics, Speech, and SignalProcessing. ASSP-35, No. 3 (1987). In the case of a simple back-offwhere P(c|ab)≈P(c|b), the model is automatically maintained because thebigrams and trigrams are already represented. When the back-off model ismore complex, such as in the model described in Katz, in someembodiments, the back-off model is not updated for every newly addedsentence s in order to reduce computational overhead of maintaining theback-off model, as described in Sethy, Abhinav, Panayiotis G. Georgiou,Bhuvana Ramabhadran, and Shrikanth Narayanan. An Iterative RelativeEntropy Minimization-Based Data Selection Approach for n-Gram ModelAdaptation. IEEE Trans. on Acoustics, Speech, and Signal Processing.17:1 (2009). Moreover, when determining whether to add a given sentences from the out-of-domain training data to Q, the language model LM of Qdoes not need to perfectly accurately correspond to the details of theback-off model, discounting (e.g., reducing frequencies of n-grams inthe language model, see Katz), and the number of sentences M in Q (aspreviously seen, for large M, ln(M)≈ln(M+1).

According to another embodiment of the present invention, the sentencescan be represented as a bag-of-trigrams {u, cnt}. FIG. 7D is a flowchartof a method 820′ for evaluating a sentence from out-of-medium trainingdata represented as bags-of-trigrams according to one embodiment of thepresent invention. Given a sentence s from the out-of-domain trainingdata, in operation 821′, the LM training module 44AA divides thesentence s into a plurality of trigrams {u}. In operation 823′, the LMtraining module 44AA counts the number of trigrams u in {u}, where thecount may be referred to herein as M. In operation 825′, the LM trainingmodule 44AA selects a next trigram u from the trigrams {u}, and inoperation 827′, the LM training module 44AA updates the entropy Ĥaccording to:

$\left. \hat{H}\leftarrow{\hat{H} - {{C(u)}\left\lbrack {{\ln\frac{u + 1}{u}} + {\ln\frac{M}{M + 1}}} \right\rbrack}} \right. = {\cong {\hat{H} + {{C(u)}\ln\frac{u + 1}{u}}}}$where M is the number of trigrams in Q.

In operation 829′, the LM training module 44AA determines whether thereare more trigrams u in the plurality of trigrams {u} to process. IF so,then the flow returns to operation 825′, where a next trigram u isselected from the trigrams {u}. If not, then the computed cross-entropyĤ is output in operation 831′.

In addition, in some embodiments of the present invention, trigramsappearing less than a threshold c number of times are replaced with atoken, e.g., “<UNK>.”

In both the n-gram model and the bag-of-trigrams model, the set Q isinitialized with data from P (e.g., a small set of sentences selectedrandomly from P) so that the above approximation of ln(M) ln(M+1) willhold. In addition, a re-computation of the maximum likelihood estimatesof the n-grams can be automatically triggered as the size of Q increases(e.g., each time Q doubles in size).

In operation 200, the LM training module 44AA may also combine all ofthe organization-specific training material (e.g., the “program,” theout-of-medium material, and the in-medium speech recognition transcriptdata). The term “source document” may be used herein to refer toorganization-specific data extracted from one of the data sources oforganization-specific training data (e.g., the in-medium trainingmaterial, the program, or the out-of-medium training material). Forexample, each email, chat session, or voice interaction will beconsidered a separate source document. For time-zero customization, eachemail, chat session, web page, forum interaction, and the like will beconsidered a separate source document. The program data may beconsidered as a separate document (e.g., the concatenation of all of theterms of interest to the organization). In some embodiments of thepresent invention, all of the organization-specific training material isconcatenated into a single document.

Once the organization specific training data are extracted in operation200, in some embodiments of the present invention, the LM trainingmodule 44AA combines the organization specific training data (e.g., theout-of-medium training data and, if available, in-medium training data)with the global training data. One way to introduce theorganization-specific training material into the language model would beto merely concatenate the organization-specific training material ontothe global training material. However, because the organization-specifictraining material is generally much smaller than the global trainingmaterial (e.g., by multiple orders of magnitude such as thousands ofentries in the customer training material versus millions of entries inthe global training material), the organization-specific trainingmaterial may have little to no impact on the resulting language model.

Another way to add the organization-specific training material would beto generate a separate language model using the organization-specifictraining material and to scale values from the global language model bya weight (e.g., a weight a, where 0≤α≤1). In particular, the selectedsentences are added with weights that fit the in-medium language modeldistribution such that likely sentences are added with higher weightthan less likely sentences. To do so, the sentences are replicated inproportion to their corresponding in-medium language model scores.

In operation 300, the LM training module 44AA compares the globalin-medium training material (e.g., transcripts of interactions fromother contact centers) with the organization specific training data tocompute a plurality of similarities. The global in-medium trainingmaterial may include transcripts of voice interactions between agents ofvarious contact centers and customers. These contact centers aregenerally contact centers of other organizations that may be indifferent business verticals. In one embodiment, each transcript in theglobal in-medium training material may be referred to as a separatedocument. In other embodiments, individual speaking turns (e.g., thefull text from when one party starts speaking until another party beginsspeaking), paragraphs, or sentences may be referred to as separatedocuments.

As such, the similarities reflect the similarity between the words andphrases found in various portions of the global in-medium trainingmaterial and the organization-specific training material.

According to one embodiment of the present invention, the similarity (or“relatedness”) of the global in-medium training material is representedby a similarity matrix. In more detail, assuming that there are N globalin-medium training documents labeled {doc₁, doc₂, . . . , doc_(N)} andthat there are M source documents in the organization-specific trainingmaterial, labeled {src₁, src₂, . . . , src_(M)}, then the similaritymatrix may be an M×N matrix, where each row of the matrix contains therelatedness between the source document src and each of the N globalin-medium training documents doc. A portion of one example similaritymatrix is shown below:

     Doc  1       Doc  N   $\begin{matrix}{src}_{1} \\{src}_{2} \\\vdots \\\vdots \\{src}_{M}\end{matrix}\begin{pmatrix}0.5 & 0.22 & \ldots & 0.47 \\\vdots & \vdots & \; & \vdots \\\vdots & \vdots & \; & \vdots \\0.1 & 0.9 & \ldots & 0.7 \\0.27 & 0.4 & \ldots & 0.18\end{pmatrix}_{MXN}$

The relatedness can be defined by a similarity or distance metric sim(i,j), which represents a similarity between an i-th source documentsrc_(i) and a j-th in-domain training document doc_(j). In oneembodiment of the present invention, the similarity of two documents iscomputed using the term frequency-inverse document frequency (tf-idf)metric, as described in, for example, Manning, C. D.; Raghavan, P.;Schutze, H. “Scoring, term weighting, and the vector space model”.Introduction to Information Retrieval (2008) and Ramos, J. “Using tf-idfto determine word relevance in document queries” First InternationalConference on Machine Learning (2003), the entire disclosures of whichare hereby incorporated by reference. Each document may be representedusing a tf-idf vector, and the similarity between two documents (e.g., asource document src and a global training material document doc) can beobtained by computing the cosine distance between the vectors of therespective documents. In some embodiments, the Apache Lucene softwarecan be used to compute the similarity between documents in this way.

In operation 400, the LM training module 44AA sorts the global in-mediumtraining data by similarity to the organization specific training data,where the sorting assists in the partition of the training data inoperation 500. In one embodiment, the LM training module 44AA uses thesimilarity matrix computed in operation 300 to compute a vector(ovs_(1×N)), which represents the overall relatedness or similarity ofeach of the global in-medium documents doc to the full set of Morganization source documents src by computing an average (e.g., mean)score of each column of the matrix (e.g., by averaging the columns inthe matrix):

${ovs}_{1 \times N} = \left( {\frac{1}{M}{\sum\limits_{i = 1}^{M}{{{sim}\left( {i,1} \right)}\mspace{14mu}\ldots\mspace{14mu}\ldots\mspace{14mu}\frac{1}{M}{\sum\limits_{i = 1}^{M}{{sim}\left( {i,N} \right)}}}}} \right)$

Each element in the vector ovs represents the similarity of a documentof the global in-medium documents to the set of organization-specificdata.

In some embodiments of the present invention, a single computation canbe performed to arrive at the vector ovs. In more detail, all of thesource documents src can be concatenated into a single company-specificsource document SRC_(ALL)=src₁+src₂+ . . . +src_(M), where the “+”operator stands for concatenation. In this embodiment, in operation 400,the LM training module 44AA computes the similarity between each of theglobal in-medium documents and the combined single company-specificsource document SRC_(ALL):ovs_(1×N)=(sim(SRC_(ALL),1)sim(SRC_(ALL),2) . . . sim(SRC_(ALL) ,N))

The LM training module 44AA then sorts the overall similarity vector ovsby similarity to the organization-specific data, as shown, for example,in Table 1 below:

TABLE 1 Doc_(j) Similarity Doc₁₃₅₅ 0.997 Doc₁₄₂₀ 0.94 . . . . . .Doc₂₇₅₀ 0.0087 Doc₇₅₀₄ 0.001

In operation 500, the LM training module 44AA partitions the sortedglobal in-medium documents can be divided into P partitions, accordingto their similarity to the organization-specific data. In oneembodiment, the number of partitions P is selected empirically based ona test set of the data. In another embodiment, a silhouette score isused to set the number of partitions, where the average silhouette of adata instance is a measure of how closely it is matched to data withinits cluster and how loosely it is matched to data of the neighboringclusters (see, e.g., R. Lleti; M. C. Ortiz; L. A. Sarabia; M. S. Sanchez(2004). “Selecting Variables for k-Means Cluster Analysis by Using aGenetic Algorithm that Optimises the Silhouettes”. Analytica ChimicaActa. 515: 87-100). According to one embodiment, the K-means techniqueis used to partition the data into the P partitions accordance withsimilarity, but embodiments of the present invention are not limitedthereto. For example, in other embodiments of the present invention, afixed set of one or more threshold similarities are used to partitionthe data (for example, documents having similarity greater than athreshold value, e.g., 0.95 may be in a first partition, and documentshaving similarity greater than 0.90 and less than or equal to 0.95 maybe assigned to the second partition). In some situations, the partitionsare the same size. In other situations, the partitions are not the samesize.

Therefore, the first partition will contain the global in-mediumdocuments that are most similar to the organization specific trainingdata, and the P-th partition will contain the least similar documents,as shown, for example, in Table 2:

TABLE 2 Partition Doc_(j) Similarity 1 Doc₁₃₅₅ 0.997 Doc₁₄₂₀ 0.94 . . .. . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . P Doc₇₅₀₂0.0088 Doc₂₇₅₀ 0.0087 Doc₇₅₀₄ 0.001

In operation 600, the LM training module 44AA weights the partitions inaccordance with their similarities. The LM training module 44AA assignseach of the P partitions a corresponding desired weight w_(d), so thatthe first partition has the highest desired weight w_(d1), and the lastpartition has the lowest weight. In one embodiment of the presentinvention, the weights decrease exponentially:

${weight}_{i} = \frac{{weight}_{1}}{\alpha^{i - 1}}$

For example, as shown in Table 3 using an exponentially decreasingweight:

TABLE 3 Example: Desired weight₁ = 1 Partition weight w_(d) α = 1; P = 41 w_(d1) 1 2$w_{d\; 2} = {\frac{w_{d\; 1}}{\alpha^{2 - 1}} = \frac{w_{d\; 1}}{\alpha^{1}}}$½ 3$w_{d\; 3} = {\frac{w_{d\; 1}}{\alpha^{3 - 1}} = \frac{w_{d\; 1}}{\alpha^{2}}}$¼ 4$w_{d\; 4} = {\frac{w_{d\; 1}}{\alpha^{4 - 1}} = \frac{w_{d\; 1}}{\alpha^{3}}}$⅛

FIG. 8 is a flowchart of a method according to one embodiment of thepresent invention for combining different training data in accordance tothe relevance of the training data based on the desired weights. Inorder for the training data to reflect the relevance of the variouspartitions P and the actual quantity of data in each of the partitionsP, in one embodiment of the present invention, the data in the partitionare scaled based weights, as described in more detail below.

Table 4, below, shows a simple example with two corpora (e.g.,documents):

TABLE 4 Corpus1 n-gram Corpus 2 n-gram n-gram count count A 17 0 a b 190 c d e 0 34 . . . . . . . . . y z 8 5 Total 44 39In the above example of Table 4, corpus 1 has a total count of 44 andcorpus 2 has a total count of 39. (In a more typical real-world case,the global training material may have a total count in the millions, andthe organization-specific training material may have a count in thethousands).

During the interpolation, to bring the weight of corpus 2 closer to adesired weight w_(d) relative to corpus 1, the counts of the n-grams ofcorpus 2 can be scaled by a factor f to the desired weight w_(d). Thecurrent weight w of corpus 2 is:

$w = \frac{{corpus}\mspace{14mu} 2\mspace{14mu}{count}}{{corpus}\mspace{14mu} 1\mspace{14mu}{count}}$then if every count in Corpus 2 is multiplied by a factor f where:

$f = \frac{w_{d}}{w}$then a different set of counts is produced for Corpus 2 where Corpus 2has the desired weight w_(d) relative to Corpus 1. In the above exampleof Table 4:

$w_{2} = {\frac{{corpus}\mspace{14mu} 2\mspace{14mu}{count}}{{corpus}\mspace{14mu} 1\mspace{14mu}{count}} = \frac{39}{44}}$Assuming, for example, that the desired weight w_(d2) of corpus 2 is ½,then the factor f₂ for corpus 2 is:

$f_{2} = {\frac{w_{d\; 2}}{w_{2}} = {\frac{1/2}{39/44} = {\frac{44}{39 \times 2} = \frac{44}{78}}}}$

Generally, the size of the organization specific training data issmaller than the size of the global training data (e.g., thousands ofsentences versus millions of sentences). In addition, the desired weightw_(d) for each of the partitions P of the global training data isgenerally smaller than the weight of the organization specific trainingdata.

As such, each of the documents in the P partitions can be considered tobe a different corpus, where each of the P partitions has acorresponding desired weight (w_(d1), . . . , w_(dP)).

As shown in FIG. 8, in one embodiment of the present invention, the LMtraining module 44AA interpolates the various documents of the globalin-medium training data with the organization specific training data byscaling 620 the counts of the n-grams in the documents. In operation621, the LM training module 44AA selects a reference document to scalethe counts to. This reference document may be, for example, theaggregated document of the organization specific training data, or maybe a typical document of the organization specific training data, or maybe an arbitrary or representative document of the first partition. Inoperation 623, the LM training module 44AA identifies the set ofdocuments {d} to be scaled in the training data. In operation 625, theLM training module 44AA selects a document d from the set of documents{d}, and identifies the desired weight w_(d) of the document d based onwhich partition includes the document d. In operation 627, the LMtraining module 44AA computes a current weight w for the document bydividing the total count of the document d divided by the total count ofthe reference document, in operation 629, identifies the desired weightw_(d) of the partition containing the document d, and, in operation 631,calculates the factor f for the document d based on the calculatedcurrent weight w and the desired weight w_(d) of the partition thatincludes the document d. In operation 633, the LM training module 44AAscales the counts of the n-grams of document d by the factor f and, inoperation 635, adds the scaled data to an output customized data set. Inoperation 637, the LM training module 44AA determines whether there aremore documents d to process in the set of documents {d}. If so, then theflow returns to operation 625 to select a next document d. If not, then,in operation 639, the LM training module 44AA outputs the customizedtraining data to train a language model.

As such, the counts of n-grams in each of the documents doc can bescaled in accordance with the corresponding factor f for the document dto select customized data for training the language model. (In theexample of Table 3 above, the weights of each partition correspond totheir desired weights w_(d) in the final language model.)

In operation 700, the LM training module 44AA trains a language modelbased on the customized training data, which includes the weightedglobal training material as well as the organization-specific trainingmaterial. Standard algorithms for computing a language model can proceedas usual using, for example, an appropriate discounting method and anappropriate back-off method.

The LM training module 44AA outputs the customized language model, whichhas been customized in accordance with organization-specific trainingmaterial. The LM training module 44AA can supply the language modelgenerated in operation 700 to the decoder module 44B, which uses thelanguage model to convert input audio into text output (speechrecognition output or LVCSR output). When using the customized languagemodel, the decoder 44B of the speech recognition module 44 may be ableto decode the input audio from the organization's voice interactionswith higher accuracy because the language model is customized based onthe organization-specific data.

The speech recognition output generated by the decoder 44B can be made apart of the on-going organization-specific training material. In moredetail, the LM training module 44AA can use the speech recognitionoutput generated by the decoder as additional organization-specific datathat can be used to calculate the weights applied to the partitions ofthe global in-medium training data, and that can also be used as part ofthe data supplied to the language model along with the global in-mediumtraining data.

In some embodiments of the present invention, in order to keep thelanguage model up-to-date with phrases and events encountered by theorganization, the process of generating a new language model based onthe new data (e.g., from the speech recognition output of the decoder44B) can be repeated at intervals (e.g., for every 50 hours of speechrecognition output results). This “online” customization can be used tomaintain a customized language model that is directed to the topics ofinterest and the current trends of phrases spoken in interactions of thetarget organization.

Computing Devices

As described herein, various applications and aspects of the presentinvention may be implemented in software, firmware, hardware, andcombinations thereof. When implemented in software, the software mayoperate on a general purpose computing device such as a server, adesktop computer, a tablet computer, a smartphone, personal digitalassistant, or an embedded system such as a computer system embedded in adevice to create an internet-of-things (IoT) device. Such a generalpurpose computer includes a general purpose processor and memory.

Each of the various servers, controllers, switches, gateways, engines,and/or modules (collectively referred to as servers) in theafore-described figures may be a process or thread, running on one ormore processors, in one or more computing devices 1500 (e.g., FIG. 9A,FIG. 9B), executing computer program instructions and interacting withother system components for performing the various functionalitiesdescribed herein. The computer program instructions are stored in amemory which may be implemented in a computing device using a standardmemory device, such as, for example, a random access memory (RAM). Thecomputer program instructions may also be stored in other non-transitorycomputer readable media such as, for example, a CD-ROM, flash drive, orthe like. Also, a person of skill in the art should recognize that acomputing device may be implemented via firmware (e.g., anapplication-specific integrated circuit), hardware, or a combination ofsoftware, firmware, and hardware. A person of skill in the art shouldalso recognize that the functionality of various computing devices maybe combined or integrated into a single computing device, or thefunctionality of a particular computing device may be distributed acrossone or more other computing devices without departing from the scope ofthe exemplary embodiments of the present invention. A server may be asoftware module, which may also simply be referred to as a module. Theset of modules in the contact center may include servers, and othermodules.

The various servers may be located on a computing device on-site at thesame physical location as the agents of the contact center or may belocated off-site (or in the cloud) in a geographically differentlocation, e.g., in a remote data center, connected to the contact centervia a network such as the Internet. In addition, some of the servers maybe located in a computing device on-site at the contact center whileothers may be located in a computing device off-site, or serversproviding redundant functionality may be provided both via on-site andoff-site computing devices to provide greater fault tolerance. In someembodiments of the present invention, functionality provided by serverslocated on computing devices off-site may be accessed and provided overa virtual private network (VPN) as if such servers were on-site, or thefunctionality may be provided using a software as a service (SaaS) toprovide functionality over the internet using various protocols, such asby exchanging data using encoded in extensible markup language (XML) orJavaScript Object notation (JSON).

FIG. 9A-FIG. 9B depict block diagrams of a computing device 1500 as maybe employed in exemplary embodiments of the present invention. Eachcomputing device 1500 includes a central processing unit 1521 and a mainmemory unit 1522. As shown in FIG. 9A, the computing device 1500 mayalso include a storage device 1528, a removable media interface 1516, anetwork interface 1518, an input/output (I/O) controller 1523, one ormore display devices 1530 c, a keyboard 1530 a and a pointing device1530 b, such as a mouse. The storage device 1528 may include, withoutlimitation, storage for an operating system and software. As shown inFIG. 9B, each computing device 1500 may also include additional optionalelements, such as a memory port 1503, a bridge 1570, one or moreadditional input/output devices 1530 d, 1530 e and a cache memory 1540in communication with the central processing unit 1521. The input/outputdevices 1530 a, 1530 b, 1530 d, and 1530 e may collectively be referredto herein using reference numeral 1530.

The central processing unit 1521 is any logic circuitry that responds toand processes instructions fetched from the main memory unit 1522. Itmay be implemented, for example, in an integrated circuit, in the formof a microprocessor, microcontroller, or graphics processing unit (GPU),or in a field-programmable gate array (FPGA) or application-specificintegrated circuit (ASIC). The main memory unit 1522 may be one or morememory chips capable of storing data and allowing any storage locationto be directly accessed by the central processing unit 1521. As shown inFIG. 9A, the central processing unit 1521 communicates with the mainmemory 1522 via a system bus 1550. As shown in FIG. 9B, the centralprocessing unit 1521 may also communicate directly with the main memory1522 via a memory port 1503.

FIG. 9B depicts an embodiment in which the central processing unit 1521communicates directly with cache memory 1540 via a secondary bus,sometimes referred to as a backside bus. In other embodiments, thecentral processing unit 1521 communicates with the cache memory 1540using the system bus 1550. The cache memory 1540 typically has a fasterresponse time than main memory 1522. As shown in FIG. 9A, the centralprocessing unit 1521 communicates with various I/O devices 1530 via thelocal system bus 1550. Various buses may be used as the local system bus1550, including a Video Electronics Standards Association (VESA) Localbus (VLB), an Industry Standard Architecture (ISA) bus, an ExtendedIndustry Standard Architecture (EISA) bus, a MicroChannel Architecture(MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Extended(PCI-X) bus, a PCI-Express bus, or a NuBus. For embodiments in which anI/O device is a display device 1530 c, the central processing unit 1521may communicate with the display device 1530 c through an AdvancedGraphics Port (AGP). FIG. 9B depicts an embodiment of a computer 1500 inwhich the central processing unit 1521 communicates directly with I/Odevice 1530 e. FIG. 9B also depicts an embodiment in which local bussesand direct communication are mixed: the central processing unit 1521communicates with I/O device 1530 d using a local system bus 1550 whilecommunicating with I/O device 1530 e directly.

A wide variety of I/O devices 1530 may be present in the computingdevice 1500. Input devices include one or more keyboards 1530 a, mice,trackpads, trackballs, microphones, and drawing tablets. Output devicesinclude video display devices 1530 c, speakers, and printers. An I/Ocontroller 1523, as shown in FIG. 9A, may control the I/O devices. TheI/O controller may control one or more I/O devices such as a keyboard1530 a and a pointing device 1530 b, e.g., a mouse or optical pen.

Referring again to FIG. 9A, the computing device 1500 may support one ormore removable media interfaces 1516, such as a floppy disk drive, aCD-ROM drive, a DVD-ROM drive, tape drives of various formats, a USBport, a Secure Digital or COMPACT FLASH™ memory card port, or any otherdevice suitable for reading data from read-only media, or for readingdata from, or writing data to, read-write media. An I/O device 1530 maybe a bridge between the system bus 1550 and a removable media interface1516.

The removable media interface 1516 may for example be used forinstalling software and programs. The computing device 1500 may furtherinclude a storage device 1528, such as one or more hard disk drives orhard disk drive arrays, for storing an operating system and otherrelated software, and for storing application software programs.Optionally, a removable media interface 1516 may also be used as thestorage device. For example, the operating system and the software maybe run from a bootable medium, for example, a bootable CD.

In some embodiments, the computing device 1500 may include or beconnected to multiple display devices 1530 c, which each may be of thesame or different type and/or form. As such, any of the I/O devices 1530and/or the I/O controller 1523 may include any type and/or form ofsuitable hardware, software, or combination of hardware and software tosupport, enable or provide for the connection to, and use of, multipledisplay devices 1530 c by the computing device 1500. For example, thecomputing device 1500 may include any type and/or form of video adapter,video card, driver, and/or library to interface, communicate, connect,or otherwise use the display devices 1530 c. In one embodiment, a videoadapter may include multiple connectors to interface to multiple displaydevices 1530 c. In other embodiments, the computing device 1500 mayinclude multiple video adapters, with each video adapter connected toone or more of the display devices 1530 c. In some embodiments, anyportion of the operating system of the computing device 1500 may beconfigured for using multiple display devices 1530 c. In otherembodiments, one or more of the display devices 1530 c may be providedby one or more other computing devices, connected, for example, to thecomputing device 1500 via a network. These embodiments may include anytype of software designed and constructed to use the display device ofanother computing device as a second display device 1530 c for thecomputing device 1500. One of ordinary skill in the art will recognizeand appreciate the various ways and embodiments that a computing device1500 may be configured to have multiple display devices 1530 c.

A computing device 1500 of the sort depicted in FIG. 9A-FIG. 9B mayoperate under the control of an operating system, which controlsscheduling of tasks and access to system resources. The computing device1500 may be running any operating system, any embedded operating system,any real-time operating system, any open source operating system, anyproprietary operating system, any operating systems for mobile computingdevices, or any other operating system capable of running on thecomputing device and performing the operations described herein.

The computing device 1500 may be any workstation, desktop computer,laptop or notebook computer, server machine, handheld computer, mobiletelephone or other portable telecommunication device, media playingdevice, gaming system, mobile computing device, or any other type and/orform of computing, telecommunications or media device that is capable ofcommunication and that has sufficient processor power and memorycapacity to perform the operations described herein. In someembodiments, the computing device 1500 may have different processors,operating systems, and input devices consistent with the device.

In other embodiments the computing device 1500 is a mobile device, suchas a Java-enabled cellular telephone or personal digital assistant(PDA), a smart phone, a digital audio player, or a portable mediaplayer. In some embodiments, the computing device 1500 includes acombination of devices, such as a mobile phone combined with a digitalaudio player or portable media player.

As shown in FIG. 9C, the central processing unit 1521 may includemultiple processors P1, P2, P3, P4, and may provide functionality forsimultaneous execution of instructions or for simultaneous execution ofone instruction on more than one piece of data. In some embodiments, thecomputing device 1500 may include a parallel processor with one or morecores. In one of these embodiments, the computing device 1500 is ashared memory parallel device, with multiple processors and/or multipleprocessor cores, accessing all available memory as a single globaladdress space. In another of these embodiments, the computing device1500 is a distributed memory parallel device with multiple processorseach accessing local memory only. In still another of these embodiments,the computing device 1500 has both some memory which is shared and somememory which may only be accessed by particular processors or subsets ofprocessors. In still even another of these embodiments, the centralprocessing unit 1521 includes a multicore microprocessor, which combinestwo or more independent processors into a single package, e.g., into asingle integrated circuit (IC). In one exemplary embodiment, depicted inFIG. 9D, the computing device 1500 includes at least one centralprocessing unit 1521 and at least one graphics processing unit 1521′.

In some embodiments, a central processing unit 1521 provides singleinstruction, multiple data (SIMD) functionality, e.g., execution of asingle instruction simultaneously on multiple pieces of data. In otherembodiments, several processors in the central processing unit 1521 mayprovide functionality for execution of multiple instructionssimultaneously on multiple pieces of data (MIMD). In still otherembodiments, the central processing unit 1521 may use any combination ofSIMD and MIMD cores in a single device.

A computing device may be one of a plurality of machines connected by anetwork, or it may include a plurality of machines so connected. FIG. 9Eshows an exemplary network environment. The network environment includesone or more local machines 1502 a, 1502 b (also generally referred to aslocal machine(s) 1502, client(s) 1502, client node(s) 1502, clientmachine(s) 1502, client computer(s) 1502, client device(s) 1502,endpoint(s) 1502, or endpoint node(s) 1502) in communication with one ormore remote machines 1506 a, 1506 b, 1506 c (also generally referred toas server machine(s) 1506 or remote machine(s) 1506) via one or morenetworks 1504. In some embodiments, a local machine 1502 has thecapacity to function as both a client node seeking access to resourcesprovided by a server machine and as a server machine providing access tohosted resources for other clients 1502 a, 1502 b. Although only twoclients 1502 and three server machines 1506 are illustrated in FIG. 9E,there may, in general, be an arbitrary number of each. The network 1504may be a local-area network (LAN), e.g., a private network such as acompany Intranet, a metropolitan area network (MAN), or a wide areanetwork (WAN), such as the Internet, or another public network, or acombination thereof.

The computing device 1500 may include a network interface 1518 tointerface to the network 1504 through a variety of connectionsincluding, but not limited to, standard telephone lines, local-areanetwork (LAN), or wide area network (WAN) links, broadband connections,wireless connections, or a combination of any or all of the above.Connections may be established using a variety of communicationprotocols. In one embodiment, the computing device 1500 communicateswith other computing devices 1500 via any type and/or form of gateway ortunneling protocol such as Secure Socket Layer (SSL) or Transport LayerSecurity (TLS). The network interface 1518 may include a built-innetwork adapter, such as a network interface card, suitable forinterfacing the computing device 1500 to any type of network capable ofcommunication and performing the operations described herein. An I/Odevice 1530 may be a bridge between the system bus 1550 and an externalcommunication bus.

According to one embodiment, the network environment of FIG. 9E may be avirtual network environment where the various components of the networkare virtualized. For example, the various machines 1502 may be virtualmachines implemented as a software-based computer running on a physicalmachine. The virtual machines may share the same operating system. Inother embodiments, different operating system may be run on each virtualmachine instance. According to one embodiment, a “hypervisor” type ofvirtualization is implemented where multiple virtual machines run on thesame host physical machine, each acting as if it has its own dedicatedbox. Of course, the virtual machines may also run on different hostphysical machines.

Other types of virtualization is also contemplated, such as, forexample, the network (e.g. via Software Defined Networking (SDN)).Functions, such as functions of the session border controller and othertypes of functions, may also be virtualized, such as, for example, viaNetwork Functions Virtualization (NFV).

While the present invention has been described in connection withcertain exemplary embodiments, it is to be understood that the inventionis not limited to the disclosed embodiments, but, on the contrary, isintended to cover various modifications and equivalent arrangementsincluded within the spirit and scope of the appended claims, andequivalents thereof.

While the present invention has been described in connection withcertain exemplary embodiments, it is to be understood that the inventionis not limited to the disclosed embodiments, but, on the contrary, isintended to cover various modifications and equivalent arrangementsincluded within the spirit and scope of the appended claims, andequivalents thereof.

What is claimed is:
 1. A method for performing voice analytics oninteractions with an organization, comprising: training a customizedlanguage model for the organization by: receiving, by a speechrecognition engine, organization-specific training data and generictraining data; computing, by the speech recognition engine, a pluralityof similarities between the generic training data and theorganization-specific training data; assigning, by the speechrecognition engine, a plurality of weights to the generic training datathrough partitioning the generic training data into a plurality ofpartitions in accordance with the computed similarities wherein thecomputed similarities comprise a fixed set of one or more thresholdsimilarities, associating a partition similarity with each of thepartitions, the partition similarity corresponding to the averagesimilarity of the data in the partition, and assigning a desired weightto each partition, the desired weight corresponding to the partitionsimilarity of the partition; combining, by the speech recognitionengine, the generic training data with the organization-specifictraining data in accordance with the weights to generate customizedtraining data; training, by the speech recognition engine, thecustomized language model using the customized training data; andoutputting, by the speech recognition engine, the customized languagemodel, the customized language model being configured to compute alikelihood of phrases in a medium; receiving, by the speech recognitionengine, an input speech from an interaction between a customer and anagent of the organization; and performing voice analytics on thereceived input speech.
 2. The method of claim 1, wherein a silhouettescore is used to determine a number of the plurality of partitions. 3.The method of claim 1, wherein a test set of the generic training dataand the organization-specific training data empirically determine anumber of the plurality of partitions.
 4. The method of claim 1, whereink-means clustering is used to determine a number of the plurality ofpartitions.
 5. The method of claim 1, wherein the desired weight of apartition is exponentially decreasing with decreasing partitionsimilarity.
 6. The method of claim 1, wherein the training a customizedlanguage model for the organization further comprise: receivingorganization-specific in-medium data; combining theorganization-specific in-medium data with the generic training data andthe organization-specific training data to generate the customizedtraining data; and retraining the language model in accordance with thecustomized training data.
 7. The method of claim 1, wherein theorganization-specific training data comprise at least one of: in-mediumdata and out-of-medium data.
 8. The method of claim 7, wherein thein-medium data comprise speech recognition transcript text and theout-of-medium data comprise non-speech text.
 9. A voice analytics systemcomprising: a speech model training system comprising: a processor; andmemory coupled to the processor and storing instructions that, whenexecuted by the processor, cause the processor to: receiveorganization-specific training data and generic training data; compute aplurality of similarities between the generic training data and theorganization-specific training data; assign a plurality of weights tothe generic training data through partitioning the generic training datainto a plurality of partitions in accordance with the computedsimilarities wherein the computed similarities comprise a fixed set ofone or more threshold similarities, associating a partition similaritywith each of the partitions, the partition similarity corresponding tothe average similarity of the data in the partition, and assigning adesired weight to each partition, the desired weight corresponding tothe partition similarity of the partition; combine the generic trainingdata with the organization-specific training data in accordance with theweights to generate customized training data; train a customizedlanguage model using the customized training data; and output thecustomized language model, the customized language model beingconfigured to compute the likelihood of phrases in a medium; and aspeech analytics system configured to: receive an input speech from aninteraction between a customer and an agent of the organization; andperform voice analytics on the received input speech.
 10. The speechrecognition system of claim 9, wherein a silhouette score is used todetermine a number of the plurality of partitions.
 11. The speechrecognition system of claim 9, wherein a test set of the generictraining data and the organization-specific training data empiricallydetermine a number of the plurality of partitions.
 12. The speechrecognition system of claim 9, wherein k-means clustering is used todetermine a number of the plurality of partitions.
 13. The speechrecognition system of claim 9, wherein the desired weight of a partitionis exponentially decreasing with decreasing partition similarity. 14.The speech recognition system of claim 9, wherein the memory of thespeech training model system further stores instructions that, whenexecuted by the processor, cause the processor to: receiveorganization-specific in-medium data; combine the organization-specificin-medium data with the generic training data and theorganization-specific training data to generate the customized trainingdata; and retrain the language model in accordance with the customizedtraining data.
 15. The speech recognition system of claim 9, wherein theorganization-specific training data comprise at least one of: in-mediumdata and out-of-medium data.
 16. The speech recognition system of claim15, wherein the in-medium data comprise speech recognition transcripttext and the out-of-medium data comprise non-speech text.